{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "covertype-training-serving-parsing.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "1BqJHo_IGuFP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Copyright 2020 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xddnGpIuHCh1",
        "colab_type": "text"
      },
      "source": [
        "<table align=\"left\">\n",
        "  <td>\n",
        "  <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/ai-platform-samples/blob/master/notebooks/samples/predictions_logging/training_and_serving.ipynb\">\n",
        "  <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab </a>\n",
        "  </td>\n",
        "  <td> \n",
        "  <a href=\"https://github.com/GoogleCloudPlatform/ai-platform-samples/blob/master/notebooks/samples/predictions_logging/training_and_serving\">\n",
        "  <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"> View on GitHub </a>\n",
        "  </td>\n",
        " </table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6YSXq_Y36c_g",
        "colab_type": "text"
      },
      "source": [
        "# Overview\n",
        "\n",
        "This tutorial shows how to train a TensorFlow classification model, using the Keras API, and deploy it to AI Platform for online prediction. The tutorial also shows how to enable [AI Platform Prediction request-response logging](https://cloud.google.com/ai-platform/prediction/docs/online-predict#requesting_logs_for_online_prediction_requests) to BigQuery.\n",
        "\n",
        "### Dataset\n",
        "\n",
        "We use the [covertype](https://archive.ics.uci.edu/ml/datasets/covertype) from UCI Machine Learning Repository. The task is to Predict forest cover type from cartographic variables only.\n",
        "\n",
        "Note that the aim is to build and deploy a **minimal model** to showcase the AI Platform Prediction request-response **logging capabilities**.\n",
        "Such logs enables further analysis for detecting on the serving data skews.\n",
        "\n",
        "### Objective\n",
        "\n",
        "The tutorial covers the following steps:\n",
        "\n",
        "1. Prepare the data and generate metadata \n",
        "2. Train and evaluate, a TensorFlow classification model using Keras API\n",
        "3. Export the trained model as a SavedModel for serving\n",
        "4. Deploy the trained model to AI Platform Prediction \n",
        "5. Enabled request-response logging to BigQuery\n",
        "6. Parse and query logs from BigQuery\n",
        "\n",
        "### Costs \n",
        "\n",
        "This tutorial uses billable components of Google Cloud Platform (GCP):\n",
        "\n",
        "* BigQuery\n",
        "* Cloud AI Platform\n",
        "* Cloud Storage\n",
        "\n",
        "You can use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage.\n",
        "\n",
        "This example uses **TensorFlow 2.x**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s2iZLfjnNxK9",
        "colab_type": "text"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "Complete the following steps to set up your development environment, install the required packages, set up your GCP project, and authenticate your GCP account."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SthW6Y1F85SH",
        "colab_type": "text"
      },
      "source": [
        "### Set up your local development environment\n",
        "\n",
        "**If you are using Colab or AI Platform Notebooks**, your environment already meets\n",
        "all the requirements to run this notebook. You can skip this step. **Otherwise**, make sure your environment meets this notebook's requirements. The Google Cloud guide to [Setting up a Python development\n",
        "environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
        "installation guide](https://jupyter.org/install) provide detailed instructions\n",
        "for meeting these requirements"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E7VJ2GIVM5SJ",
        "colab_type": "text"
      },
      "source": [
        "### PIP Install Packages and dependencies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sm5FTgF69bll",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install -q -U tensorflow==2.1\n",
        "!pip install -U -q google-api-python-client\n",
        "!pip install -U -q pandas"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_AXP7PUBNGt5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Automatically restart kernel after installs\n",
        "import IPython\n",
        "app = IPython.Application.instance()\n",
        "app.kernel.do_shutdown(True)  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mdqP-e0xNh13",
        "colab_type": "text"
      },
      "source": [
        "### Set up your GCP project and GCS bucket\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a GCP project.](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "\n",
        "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
        "\n",
        "3. [Enable the AI Platform APIs and Compute Engine APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component)\n",
        "\n",
        "4. If you are running this notebook locally, you will need to install [Google Cloud SDK](https://cloud.google.com/sdk).\n",
        "\n",
        "\n",
        "When you deploy a model to AI Platform Prediction, you need to upload your model artifacts to a Cloud Storage bucket. You can then\n",
        "create an AI Platform model version based on these artifacts in order to serve online predictions. **Make sure that you GCS bucket already exists**. See [creating Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets).  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ngafVPJ0THwE",
        "colab_type": "text"
      },
      "source": [
        "Enter your project ID, bucket, and region bellow. Make sure to [choose a region where Cloud AI Platform services are\n",
        "available](https://cloud.google.com/ml-engine/docs/tensorflow/regions)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GufMPSFuOSiZ",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "PROJECT_ID = \"[your-project-Id]\" #@param {type:\"string\"}\n",
        "BUCKET = '[your-bucket-name]' #@param {type:\"string\"}\n",
        "REGION = 'us-central1' #@param {type:\"string\"}\n",
        "!gcloud config set project $PROJECT_ID"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8_9K14ssNRnC",
        "colab_type": "text"
      },
      "source": [
        "### Authenticate your GCP account\n",
        "\n",
        "**If you are using AI Platform Notebooks**, your environment is already\n",
        "authenticated. Skip this step.\n",
        "\n",
        "**If you are using Colab**, run the cell below and follow the instructions\n",
        "when prompted to authenticate your account via oAuth."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kWzmq-55CZfV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "try:\n",
        "  from google.colab import auth\n",
        "  auth.authenticate_user()\n",
        "  print(\"Colab user is authenticated.\")\n",
        "except: pass"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ewFHEkCaO_mW",
        "colab_type": "text"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t4pR1SEo6Tra",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import os\n",
        "import tensorflow as tf\n",
        "import pandas as pd\n",
        "from google.cloud import bigquery\n",
        "\n",
        "print(\"TF version: {}\".format(tf.__version__))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RGkiBa7eQYKm",
        "colab_type": "text"
      },
      "source": [
        "### Define constants\n",
        "\n",
        "You can change the default values for the following constants"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qIVPK2qaQYRm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "LOCAL_WORKSPACE = './workspace'\n",
        "LOCAL_DATA_DIR = os.path.join(LOCAL_WORKSPACE, 'data')\n",
        "BQ_DATASET_NAME = 'prediction_logs'\n",
        "BQ_TABLE_NAME = 'covertype_classifier_logs' \n",
        "MODEL_NAME = 'covertype_classifier'\n",
        "VERSION_NAME = 'v1' \n",
        "TRAINING_DIR = os.path.join(LOCAL_WORKSPACE, 'training')\n",
        "MODEL_DIR = os.path.join(TRAINING_DIR, 'exported_model')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xmYHPLFgEA92",
        "colab_type": "text"
      },
      "source": [
        "### Create a local workspace"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0XXOcV8BD_w9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "if tf.io.gfile.exists(LOCAL_WORKSPACE):\n",
        "  print(\"Removing previous workspace artifacts...\")\n",
        "  tf.io.gfile.rmtree(LOCAL_WORKSPACE)\n",
        "\n",
        "print(\"Creating a new workspace...\")\n",
        "tf.io.gfile.makedirs(LOCAL_WORKSPACE)\n",
        "tf.io.gfile.makedirs(LOCAL_DATA_DIR)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LLUHs-gh_FnY",
        "colab_type": "text"
      },
      "source": [
        "## 1. Dataset preparation and schema generation \n",
        "\n",
        "The dataset is preprocessed, split, and uploaded to the `gs://workshop-datasets/covertype` public GCS location. \n",
        "\n",
        "We use this version of the preprocessed dataset in this notebook. For more information, see [Cover Type Dataset](https://github.com/GoogleCloudPlatform/mlops-on-gcp/tree/master/datasets/covertype)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qDH43VPLCCsS",
        "colab_type": "text"
      },
      "source": [
        "### 1.1. Download the data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VK3t6GswFJaa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "LOCAL_TRAIN_DATA = os.path.join(LOCAL_DATA_DIR, 'train.csv') \n",
        "LOCAL_EVAL_DATA = os.path.join(LOCAL_DATA_DIR, 'eval.csv') "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pIjykELMB-Zx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!gsutil cp gs://workshop-datasets/covertype/data_validation/training/dataset.csv {LOCAL_TRAIN_DATA}\n",
        "!gsutil cp gs://workshop-datasets/covertype/data_validation/evaluation/dataset.csv {LOCAL_EVAL_DATA}\n",
        "!wc -l {LOCAL_TRAIN_DATA}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AkkFi52UUEfS",
        "colab_type": "text"
      },
      "source": [
        "View a sample of the downloaded data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MV0j0nAezO_R",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sample = pd.read_csv(LOCAL_TRAIN_DATA).head()\n",
        "sample.T"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JUjRurNblnaw",
        "colab_type": "text"
      },
      "source": [
        "### 1.2 Define metadata\n",
        "The following is metadata of the dataset, which is used to create the data input function, feature columns, and serving function. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "72rDTLhcUxj_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "HEADER = ['Elevation', 'Aspect', 'Slope','Horizontal_Distance_To_Hydrology',\n",
        "          'Vertical_Distance_To_Hydrology', 'Horizontal_Distance_To_Roadways',\n",
        "          'Hillshade_9am', 'Hillshade_Noon', 'Hillshade_3pm',\n",
        "          'Horizontal_Distance_To_Fire_Points', 'Wilderness_Area', 'Soil_Type',\n",
        "          'Cover_Type']\n",
        "\n",
        "TARGET_FEATURE_NAME = 'Cover_Type'\n",
        "\n",
        "FEATURE_LABELS = ['0', '1', '2', '3', '4', '5', '6']\n",
        "\n",
        "NUMERIC_FEATURE_NAMES = ['Aspect', 'Elevation', 'Hillshade_3pm', \n",
        "                         'Hillshade_9am', 'Hillshade_Noon', \n",
        "                         'Horizontal_Distance_To_Fire_Points',\n",
        "                         'Horizontal_Distance_To_Hydrology',\n",
        "                         'Horizontal_Distance_To_Roadways','Slope',\n",
        "                         'Vertical_Distance_To_Hydrology']\n",
        "\n",
        "CATEGORICAL_FEATURES_WITH_VOCABULARY = {\n",
        "    'Soil_Type': ['2702', '2703', '2704', '2705', '2706', '2717', '3501', '3502', \n",
        "                  '4201', '4703', '4704', '4744', '4758', '5101', '6101', '6102', \n",
        "                  '6731', '7101', '7102', '7103', '7201', '7202', '7700', '7701', \n",
        "                  '7702', '7709', '7710', '7745', '7746', '7755', '7756', '7757', \n",
        "                  '7790', '8703', '8707', '8708', '8771', '8772', '8776'], \n",
        "    'Wilderness_Area': ['Cache', 'Commanche', 'Neota', 'Rawah']\n",
        "}\n",
        "\n",
        "FEATURE_NAMES = list(CATEGORICAL_FEATURES_WITH_VOCABULARY.keys()) + NUMERIC_FEATURE_NAMES\n",
        "\n",
        "HEADER_DEFAULTS = [[0] if feature_name in NUMERIC_FEATURE_NAMES + [TARGET_FEATURE_NAME] else ['NA'] \n",
        "                   for feature_name in HEADER]\n",
        "\n",
        "NUM_CLASSES = len(FEATURE_LABELS)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "20wwieKoJvL1",
        "colab_type": "text"
      },
      "source": [
        "## 2. Model training and evaluation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6e3lk9FsJ_yO",
        "colab_type": "text"
      },
      "source": [
        "### 2.1. Implement data input pipeline"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PTX-t2HjJ_BP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "RANDOM_SEED = 19830610\n",
        "import multiprocessing\n",
        "\n",
        "def create_dataset(file_pattern, \n",
        "                  batch_size=128, num_epochs=1, shuffle=False):\n",
        "  \n",
        "    dataset = tf.data.experimental.make_csv_dataset(\n",
        "        file_pattern=file_pattern,\n",
        "        batch_size=batch_size,\n",
        "        column_names=HEADER,\n",
        "        column_defaults=HEADER_DEFAULTS,\n",
        "        label_name=TARGET_FEATURE_NAME,\n",
        "        field_delim=',',\n",
        "        header=True,\n",
        "        num_epochs=num_epochs,\n",
        "        shuffle=shuffle,\n",
        "        shuffle_buffer_size=(5 * batch_size),\n",
        "        shuffle_seed=RANDOM_SEED,\n",
        "        num_parallel_reads=multiprocessing.cpu_count(),\n",
        "        sloppy=True,\n",
        "    )\n",
        "    return dataset.cache()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T3ubvOGrUV5S",
        "colab_type": "text"
      },
      "source": [
        "The following code test reading some batches of data using the data input function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sGcaJmcG3paA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "index = 1\n",
        "for batch in create_dataset(LOCAL_TRAIN_DATA, batch_size=5, shuffle=False).take(2):\n",
        "  print(\"Batch: {}\".format(index))\n",
        "  print(\"========================\")\n",
        "  record, target = batch\n",
        "  print(\"Input features:\")\n",
        "  for key in record:\n",
        "    print(\" - {}:{}\".format(key, record[key].numpy()))\n",
        "  print(\"Target: {}\".format(target))\n",
        "  index += 1\n",
        "  print()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8s8a7bMGKLWu",
        "colab_type": "text"
      },
      "source": [
        "### 2.2. Create feature columns"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C8pZSfQPHMrK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import math\n",
        "\n",
        "def create_feature_columns():\n",
        "  feature_columns = []\n",
        "  \n",
        "  for feature_name in FEATURE_NAMES:\n",
        "    # Categorical features\n",
        "    if feature_name in CATEGORICAL_FEATURES_WITH_VOCABULARY:\n",
        "      \n",
        "      vocabulary = CATEGORICAL_FEATURES_WITH_VOCABULARY[feature_name]\n",
        "      vocab_size = len(vocabulary)\n",
        "      \n",
        "      # Create embedding column for categotical feature column with vocabulary\n",
        "      embedding_feature_column = tf.feature_column.embedding_column(\n",
        "          categorical_column = tf.feature_column.categorical_column_with_vocabulary_list(\n",
        "              key=feature_name,\n",
        "              vocabulary_list=vocabulary), dimension=int(math.sqrt(vocab_size) + 1))\n",
        "            \n",
        "      feature_columns.append(embedding_feature_column)\n",
        "\n",
        "    # Numeric features\n",
        "    else:\n",
        "      numeric_column = tf.feature_column.numeric_column(feature_name)\n",
        "      feature_columns.append(numeric_column)\n",
        "\n",
        "  return feature_columns\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pLvEmxgvUgKd",
        "colab_type": "text"
      },
      "source": [
        "The following code tests the feature columns to be created."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N8XDvPsq8-mX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "feature_columns = create_feature_columns()\n",
        "\n",
        "for column in feature_columns:\n",
        "  print(column)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F0pAN0jkKNmm",
        "colab_type": "text"
      },
      "source": [
        "### 2.3. Create and compile the model\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E98fiPtQKTp2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def create_model(params):\n",
        "\n",
        "  feature_columns = create_feature_columns()\n",
        "  \n",
        "  layers = []\n",
        "  layers.append(tf.keras.layers.DenseFeatures(feature_columns))\n",
        "  for units in params.hidden_units:\n",
        "    layers.append(tf.keras.layers.Dense(units=units, activation='relu'))\n",
        "    layers.append(tf.keras.layers.BatchNormalization())\n",
        "    layers.append(tf.keras.layers.Dropout(rate=params.dropout))\n",
        "  \n",
        "  layers.append(tf.keras.layers.Dense(units=NUM_CLASSES, activation='softmax'))\n",
        "  \n",
        "  model = tf.keras.Sequential(layers=layers, name='classifier')\n",
        "    \n",
        "  adam_optimzer = tf.keras.optimizers.Adam(learning_rate=params.learning_rate)\n",
        "\n",
        "  model.compile(\n",
        "        optimizer=adam_optimzer, \n",
        "        loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False), \n",
        "        metrics=[tf.keras.metrics.SparseCategoricalAccuracy()], \n",
        "        loss_weights=None,\n",
        "        sample_weight_mode=None, \n",
        "        weighted_metrics=None, \n",
        "    )\n",
        "\n",
        "  return model  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MjzFWQnSJFse",
        "colab_type": "text"
      },
      "source": [
        "### 2.4. Train and evaluate experiment"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H9_yXPvF2Hkd",
        "colab_type": "text"
      },
      "source": [
        "#### Experiment"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F9iW5NumKWWb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def run_experiment(model, params):\n",
        "\n",
        "  # TensorBoard callback\n",
        "  LOG_DIR = os.path.join(TRAINING_DIR, 'logs')\n",
        "  tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=LOG_DIR)\n",
        "\n",
        "  # early stopping callback\n",
        "  earlystopping_callback = tf.keras.callbacks.EarlyStopping(\n",
        "      monitor='val_sparse_categorical_accuracy', \n",
        "      patience=3, \n",
        "      restore_best_weights=True\n",
        "  )\n",
        "\n",
        "  callbacks = [\n",
        "        tensorboard_callback,\n",
        "        earlystopping_callback]\n",
        "\n",
        "  # train dataset\n",
        "  train_dataset = create_dataset(\n",
        "      LOCAL_TRAIN_DATA,\n",
        "      batch_size=params.batch_size,\n",
        "      shuffle=True)\n",
        "    \n",
        "  # eval dataset\n",
        "  eval_dataset = create_dataset(\n",
        "      LOCAL_EVAL_DATA,\n",
        "      batch_size=params.batch_size)\n",
        "    \n",
        "  # Prep training directory\n",
        "  if tf.io.gfile.exists(TRAINING_DIR):\n",
        "    print(\"Removing previous training artefacts...\")\n",
        "    tf.io.gfile.rmtree(TRAINING_DIR)\n",
        "\n",
        "  print(\"Creating training directory...\")\n",
        "  tf.io.gfile.mkdir(TRAINING_DIR)\n",
        "\n",
        "  print(\"Experiment started...\")\n",
        "  print(\".......................................\")\n",
        "  \n",
        "  # Run train and evaluate.\n",
        "  history = model.fit(\n",
        "    x=train_dataset, \n",
        "    epochs=params.epochs, \n",
        "    callbacks=callbacks,\n",
        "    validation_data=eval_dataset,\n",
        "  )\n",
        "\n",
        "  print(\".......................................\")\n",
        "  print(\"Experiment finished.\")\n",
        "  print(\"\")\n",
        "\n",
        "  return history\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PhYvbrrqJKmn",
        "colab_type": "text"
      },
      "source": [
        "#### Hyper parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zBCwS_usHvqy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Parameters():\n",
        "    pass\n",
        "\n",
        "TRAIN_DATA_SIZE = 431010\n",
        "\n",
        "params = Parameters()\n",
        "params.learning_rate = 0.01\n",
        "params.hidden_units = [128, 128]\n",
        "params.dropout = 0.15\n",
        "params.batch_size =  265\n",
        "params.steps_per_epoch = int(math.ceil(TRAIN_DATA_SIZE / params.batch_size))\n",
        "params.epochs = 10"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sf7FGOscJZ2P",
        "colab_type": "text"
      },
      "source": [
        "#### Run experiment"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ew99zH72JRjp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = create_model(params)\n",
        "example_batch, _ = list(\n",
        "    create_dataset(LOCAL_TRAIN_DATA, batch_size=2, shuffle=True).take(1))[0]\n",
        "model(example_batch)\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E0DjaL7eI8Ro",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import logging\n",
        "logger = tf.get_logger()\n",
        "logger.setLevel(logging.ERROR)\n",
        "\n",
        "history = run_experiment(model, params)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UhJ5qV0aJd0J",
        "colab_type": "text"
      },
      "source": [
        "#### Visualize training history"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ie7fi-q_JAhh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "fig, (ax1, ax2) = plt.subplots(1, 2)\n",
        "fig.set_size_inches(w=(10, 5))\n",
        "\n",
        "# Plot training & validation accuracy values\n",
        "ax1.plot(history.history['sparse_categorical_accuracy'])\n",
        "ax1.plot(history.history['val_sparse_categorical_accuracy'])\n",
        "ax1.set_title('Model accuracy')\n",
        "ax1.set(xlabel='Accuracy', ylabel='Epoch')\n",
        "\n",
        "ax1.legend(['Train', 'Eval'], loc='upper left')\n",
        "\n",
        "\n",
        "# Plot training & validation loss values\n",
        "ax2.plot(history.history['loss'])\n",
        "ax2.plot(history.history['val_loss'])\n",
        "ax2.set_title('Model loss')\n",
        "ax2.set(xlabel='Accuracy', ylabel='Epoch')\n",
        "ax2.legend(['Train', 'Eval'], loc='upper left')\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t7q4uSu1KbXv",
        "colab_type": "text"
      },
      "source": [
        "## 3. Model export for serving"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ulI3HOMLS-Zu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "MODEL_OUTPUT_KEY = 'probabilities'\n",
        "SIGNATURE_NAME = 'serving_default'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vwDylJzoUTb1",
        "colab_type": "text"
      },
      "source": [
        "### 3.1. Implement serving input receiver functions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Iijp9_mV0gNn",
        "colab_type": "text"
      },
      "source": [
        "#### Serving function\n",
        "\n",
        "We create serving input function that expects features dictionary and returns the prediction pobabilities from the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hQDJP47xnt9p",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def make_features_serving_fn(model):\n",
        "\n",
        "  @tf.function\n",
        "  def serve_features_fn(features):\n",
        "    return {MODEL_OUTPUT_KEY: model(features)}\n",
        "\n",
        "  return serve_features_fn"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lhYjXNxgjob1",
        "colab_type": "text"
      },
      "source": [
        "#### Feature spec\n",
        "\n",
        "We create feature_spec dictionary for the input features with respect to the dataset metadata"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_D577fVOKbcw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "feature_spec = {}\n",
        "for feature_name in FEATURE_NAMES:\n",
        "    if feature_name in CATEGORICAL_FEATURES_WITH_VOCABULARY:\n",
        "        feature_spec[feature_name] = tf.io.FixedLenFeature(\n",
        "            shape=(1,), dtype=tf.string)\n",
        "    else:\n",
        "        feature_spec[feature_name] = tf.io.FixedLenFeature(\n",
        "            shape=(1,), dtype=tf.float32)\n",
        "\n",
        "for key, value in feature_spec.items():\n",
        "  print(\"{}: {}\".format(key, value))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2sPh4f_tUX5n",
        "colab_type": "text"
      },
      "source": [
        "### 3.2. Export the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZlQfiluTTuW-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "features_input_signature = {\n",
        "    feature: tf.TensorSpec(shape=spec.shape, dtype=spec.dtype, name=feature)\n",
        "    for feature, spec in feature_spec.items()\n",
        "    }\n",
        "\n",
        "signatures = {        \n",
        "    SIGNATURE_NAME: make_features_serving_fn(model).get_concrete_function(\n",
        "        features_input_signature),\n",
        "    }\n",
        "\n",
        "model.save(MODEL_DIR, save_format='tf', signatures=signatures)\n",
        "print(\"Model is exported to: {}.\".format(MODEL_DIR))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7zmuFWnRU7eJ",
        "colab_type": "text"
      },
      "source": [
        "Verify the signature (inputs and outputs) of the exported model using `saved_model_cli`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fyK7Yy0CCr1f",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!saved_model_cli show --dir {MODEL_DIR} --tag_set serve --signature_def {SIGNATURE_NAME}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9AQdyoPZUaKH",
        "colab_type": "text"
      },
      "source": [
        "### 3.3. Test exported model locally"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BG9MM4mkVK5A",
        "colab_type": "text"
      },
      "source": [
        "Create a sample instance for prediction"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_gfRf20JVgn6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "instances = [\n",
        "      { \n",
        "        'Soil_Type': '7202',\n",
        "        'Wilderness_Area': 'Commanche',\n",
        "        'Aspect': 61,\n",
        "        'Elevation': 3091,\n",
        "        'Hillshade_3pm': 129,\n",
        "        'Hillshade_9am': 227,\n",
        "        'Hillshade_Noon': 223,\n",
        "        'Horizontal_Distance_To_Fire_Points': 2868,\n",
        "        'Horizontal_Distance_To_Hydrology': 134,\n",
        "        'Horizontal_Distance_To_Roadways': 0, \n",
        "        'Slope': 8, \n",
        "        'Vertical_Distance_To_Hydrology': 10,\n",
        "    }\n",
        "]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IV07vVr1uBvr",
        "colab_type": "text"
      },
      "source": [
        "Prepare the sample instance in the format expected by the model signature"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AHuRrUjvuK0R",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "def create_tf_features(instance):\n",
        " \n",
        "  new_instance = {}\n",
        "  for key, value in instance.items():\n",
        "    if key in CATEGORICAL_FEATURES_WITH_VOCABULARY:\n",
        "      new_instance[key] = tf.constant(value, dtype=tf.string)\n",
        "    else:\n",
        "      new_instance[key] = tf.constant(value, dtype=tf.float32)\n",
        "  \n",
        "  return new_instance"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wKVRSluEWCRb",
        "colab_type": "text"
      },
      "source": [
        "Load the SavedModel for prediction, and create a function that generates the prediction pobabilities from the model to return the class label with the highest probability"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RUaWFXqgu96i",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "features_predictor = tf.saved_model.load(MODEL_DIR).signatures[SIGNATURE_NAME]\n",
        "\n",
        "def local_predict(instance):\n",
        "\n",
        "  features = create_tf_features(instance)\n",
        "  probabilities = features_predictor(**features)[MODEL_OUTPUT_KEY].numpy()\n",
        "  predictions = FEATURE_LABELS[int(np.argmax(probabilities))]\n",
        "\n",
        "  return predictions"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-J6SZ6jwWHJW",
        "colab_type": "text"
      },
      "source": [
        "Predict using the local SavedModel"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N2obiJWB20BX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "local_predict(instances[0])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fY-3txmZb0kf",
        "colab_type": "text"
      },
      "source": [
        "### 3.4  Upload exported model to GCS"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E6X0wsuJb0rB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!gsutil rm -r gs://{BUCKET}/models/{MODEL_NAME}\n",
        "!gsutil cp -r {MODEL_DIR} gs://{BUCKET}/models/{MODEL_NAME}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gvVcRazvL_95",
        "colab_type": "text"
      },
      "source": [
        "## 4. Model deployment to AI Platform \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uncXaPUBdBpV",
        "colab_type": "text"
      },
      "source": [
        "### 4.1. Create model in AI Platform"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bO5a1fGLLrSv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!gcloud ai-platform models create {MODEL_NAME} \\\n",
        "  --project {PROJECT_ID} \\\n",
        "  --regions {REGION}\n",
        "\n",
        "# list the models\n",
        "!gcloud ai-platform models list --project {PROJECT_ID}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VmUPxnCZdJZ4",
        "colab_type": "text"
      },
      "source": [
        "### 4.2. Create a model version"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jW-zQi9bZzF5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!gcloud ai-platform versions create {VERSION_NAME} \\\n",
        "  --model={MODEL_NAME} \\\n",
        "  --origin=gs://{BUCKET}/models/{MODEL_NAME} \\\n",
        "  --runtime-version=2.1 \\\n",
        "  --framework=TENSORFLOW \\\n",
        "  --python-version=3.7 \\\n",
        "  --project={PROJECT_ID}\n",
        "\n",
        "# list the model versions\n",
        "!gcloud ai-platform versions list --model={MODEL_NAME} --project={PROJECT_ID}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nmFimRVrc4m8",
        "colab_type": "text"
      },
      "source": [
        "### 4.3. Test deployed model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2dGRzGf0Wcth",
        "colab_type": "text"
      },
      "source": [
        "Create a function to call the AI Platform Prediction model version"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YJJDQNMzc4xh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import googleapiclient.discovery\n",
        "\n",
        "service = googleapiclient.discovery.build('ml', 'v1')\n",
        "name = 'projects/{}/models/{}/versions/{}'.format(PROJECT_ID, MODEL_NAME, VERSION_NAME)\n",
        "print(\"Service name: {}\".format(name))\n",
        "\n",
        "def caip_predict(instances):\n",
        "  \n",
        "  serving_instances = []\n",
        "  for instance in instances:\n",
        "    serving_instances.append(\n",
        "        {key: [value] for key, value in instance.items()})\n",
        "    \n",
        "  request_body={\n",
        "      'signature_name': SIGNATURE_NAME,\n",
        "      'instances': serving_instances}\n",
        "\n",
        "  response = service.projects().predict(\n",
        "      name=name,\n",
        "      body=request_body\n",
        "\n",
        "  ).execute()\n",
        "\n",
        "  if 'error' in response:\n",
        "    raise RuntimeError(response['error'])\n",
        "\n",
        "  probability_list = [output[MODEL_OUTPUT_KEY] for output in response['predictions']]\n",
        "  classes = [FEATURE_LABELS[int(np.argmax(probabilities))] for probabilities in probability_list]\n",
        "  return classes"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y3WZZSqyWeGZ",
        "colab_type": "text"
      },
      "source": [
        "Predict using AI Platform Prediction"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lbgnmaks4UPC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "caip_predict(instances)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m11-MCflLjFG",
        "colab_type": "text"
      },
      "source": [
        "## 5. BigQuery logging dataset preparation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HmWQwyAUdwIW",
        "colab_type": "text"
      },
      "source": [
        "### 5.1. Create the BigQuery dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ne38zMEKL_Gk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "client = bigquery.Client(PROJECT_ID)\n",
        "dataset_names = [dataset.dataset_id for dataset in client.list_datasets(PROJECT_ID)]\n",
        "\n",
        "dataset = bigquery.Dataset(\"{}.{}\".format(PROJECT_ID, BQ_DATASET_NAME))\n",
        "dataset.location = \"US\"\n",
        "\n",
        "if BQ_DATASET_NAME not in dataset_names:\n",
        "  dataset = client.create_dataset(dataset)\n",
        "  print(\"Created dataset {}.{}\".format(client.project, dataset.dataset_id))\n",
        "\n",
        "print(\"BigQuery Dataset is ready.\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ieIgLE94PovQ",
        "colab_type": "text"
      },
      "source": [
        "### 5.2. Create the BigQuery table to store the logs\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ima0Dg1UWkRO",
        "colab_type": "text"
      },
      "source": [
        "#### Table schema"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TqxS6RQ0T9LG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import json\n",
        "\n",
        "table_schema_json = [\n",
        "  {\n",
        "    \"name\": \"model\", \n",
        "    \"type\": \"STRING\", \n",
        "    \"mode\": \"REQUIRED\"\n",
        "   },\n",
        "   {\n",
        "     \"name\":\"model_version\", \n",
        "     \"type\": \"STRING\", \n",
        "     \"mode\":\"REQUIRED\"\n",
        "  },\n",
        "  {\n",
        "    \"name\":\"time\", \n",
        "    \"type\": \"TIMESTAMP\", \n",
        "    \"mode\": \"REQUIRED\"\n",
        "  },\n",
        "  {\n",
        "    \"name\":\"raw_data\", \n",
        "    \"type\": \"STRING\", \n",
        "    \"mode\": \"REQUIRED\"\n",
        "  },\n",
        "  {\n",
        "    \"name\":\"raw_prediction\", \n",
        "    \"type\": \"STRING\", \n",
        "    \"mode\": \"NULLABLE\"\n",
        "  },\n",
        "  {\n",
        "    \"name\":\"groundtruth\", \n",
        "    \"type\": \"STRING\", \n",
        "    \"mode\": \"NULLABLE\"\n",
        "  },\n",
        "]\n",
        "\n",
        "json.dump(\n",
        "    table_schema_json, open('table_schema.json', 'w'))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "koE5tKouWngb",
        "colab_type": "text"
      },
      "source": [
        "#### Ceating an ingestion-time partitioned tables"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SXkoAGEiXRvj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "table = bigquery.Table(\n",
        "    \"{}.{}.{}\".format(PROJECT_ID, BQ_DATASET_NAME, BQ_TABLE_NAME))\n",
        "\n",
        "table_names = [table.table_id for table in client.list_tables(dataset)]\n",
        "\n",
        "if BQ_TABLE_NAME in table_names:\n",
        "  print(\"Deleteing BQ Table: {} ...\".format(BQ_TABLE_NAME))\n",
        "  client.delete_table(table)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QnE9GMjbeqXR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "TIME_PARTITION_EXPERIATION = int(60 * 60 * 24 * 7)\n",
        "\n",
        "!bq mk --table \\\n",
        "  --project_id={PROJECT_ID} \\\n",
        "  --time_partitioning_field=time \\\n",
        "  --time_partitioning_type=DAY \\\n",
        "  --time_partitioning_expiration={TIME_PARTITION_EXPERIATION} \\\n",
        "  {PROJECT_ID}:{BQ_DATASET_NAME}.{BQ_TABLE_NAME} \\\n",
        "  'table_schema.json'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RPFM4mW-R3y8",
        "colab_type": "text"
      },
      "source": [
        "### 5.3. Configre the AI Platform model version to enable request-response logging to BigQuery\n",
        "\n",
        "In order to enable the request-response logging to an existing AI Platform Prediction model version, you need to call the `patch` API and populate the [requestLoggingConfig](https://cloud.google.com/ai-platform/prediction/docs/online-predict#requesting_logs_for_online_prediction_requests) field."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PBkx4i9Rc51W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sampling_percentage = 1.0\n",
        "bq_full_table_name = '{}.{}.{}'.format(PROJECT_ID, BQ_DATASET_NAME, BQ_TABLE_NAME)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EarF0cmcRZN4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "logging_config = {\n",
        "    \"requestLoggingConfig\":{\n",
        "        \"samplingPercentage\": sampling_percentage,\n",
        "        \"bigqueryTableName\": bq_full_table_name\n",
        "        }\n",
        "    }\n",
        "\n",
        "service.projects().models().versions().patch(\n",
        "    name=name,\n",
        "    body=logging_config,\n",
        "    updateMask=\"requestLoggingConfig\"\n",
        "    ).execute()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kVpICSuq0Kp1",
        "colab_type": "text"
      },
      "source": [
        "### 5.4. Test request-response logging"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3XVWG6y3XWIq",
        "colab_type": "text"
      },
      "source": [
        "Send sample prediction requests to the model version on AI Platform Prediction"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "6YsfjkPtkx3o",
        "colab": {}
      },
      "source": [
        "import time\n",
        "\n",
        "for i in range(10):\n",
        "  caip_predict(instances)\n",
        "  print('.', end='')\n",
        "  time.sleep(0.1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vgi_HyXIXcdg",
        "colab_type": "text"
      },
      "source": [
        "Query the logged request-reponse entries in BigQuery"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jdlrKX0d0mQC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "query = '''\n",
        "  SELECT * FROM \n",
        "  `{}.{}` \n",
        "  WHERE model_version = '{}'\n",
        "  ORDER BY time desc\n",
        "  LIMIT {}\n",
        "'''.format(BQ_DATASET_NAME, BQ_TABLE_NAME, VERSION_NAME, 3)\n",
        "\n",
        "pd.io.gbq.read_gbq(\n",
        "    query, project_id=PROJECT_ID).T\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vK_yvIjWkPk7",
        "colab_type": "text"
      },
      "source": [
        "## 6. BigQuery logs parsing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z4drXsHAXKZC",
        "colab_type": "text"
      },
      "source": [
        "### 6.1. Generate the CREATE VIEW script"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6fTqQkXEJszV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "view_name = \"vw_\"+BQ_TABLE_NAME+\"_\"+VERSION_NAME\n",
        "\n",
        "colum_names = FEATURE_NAMES\n",
        "input_features = ', \\r\\n  '.join(colum_names)\n",
        "\n",
        "json_features_extraction = []\n",
        "for feature_name in colum_names:\n",
        "  s = \"JSON_EXTRACT(instance, '$.{}')\".format(feature_name) \n",
        "  if feature_name in NUMERIC_FEATURE_NAMES:\n",
        "    s = \"CAST({} AS NUMERIC)\".format(s)\n",
        "  s += \" AS {}\".format(feature_name)\n",
        "  json_features_extraction.append(s)\n",
        "json_features_extraction = ', \\r\\n    '.join(json_features_extraction)\n",
        "\n",
        "class_probability_pivoting = []\n",
        "for class_index, class_label in enumerate(FEATURE_LABELS):\n",
        "  s = \"CAST(MAX(IF(class_index = {}, class_probability, NULL)) as FLOAT64) as prob_{}\".format(class_index, class_label)\n",
        "  class_probability_pivoting.append(s)\n",
        "class_probability_pivoting = ', \\r\\n  '.join(class_probability_pivoting)\n",
        "\n",
        "class_prob = []\n",
        "for class_label in FEATURE_LABELS:\n",
        "  s = 'prob_{}'.format(class_label)\n",
        "  class_prob.append(s)\n",
        "\n",
        "class_prob = ', \\r\\n  '.join(class_prob)\n",
        "\n",
        "case_conditions = []\n",
        "for class_label in FEATURE_LABELS:\n",
        "  s = 'WHEN prob_max = prob_{} THEN {}'.format(class_label, class_label)\n",
        "  case_conditions.append(s)\n",
        "case_conditions = '   \\r\\n '.join(case_conditions)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "69vABhIJcqbR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sql_script = '''\n",
        "CREATE OR REPLACE VIEW @dataset_name.@view_name\n",
        "AS\n",
        "\n",
        "WITH step1\n",
        "AS\n",
        "(\n",
        "  SELECT \n",
        "    model, \n",
        "    model_version, \n",
        "    time, \n",
        "    SPLIT(JSON_EXTRACT(raw_data, '$.instances'), '}],[{') instance_list, \n",
        "    SPLIT(JSON_EXTRACT(raw_prediction, '$.predictions'), '}],[{') as prediction_list\n",
        "  FROM \n",
        "  `@project.@dataset_name.@table_name` \n",
        "  WHERE \n",
        "    model = '@model_name' AND\n",
        "    model_version = '@version'\n",
        "),\n",
        "\n",
        "step2\n",
        "AS\n",
        "(\n",
        "  SELECT\n",
        "    model, \n",
        "    model_version, \n",
        "    time, \n",
        "    REPLACE(REPLACE(instance, '[', ''),']', '') AS instance,\n",
        "    REPLACE(REPLACE(prediction, '[{\"@model_output_key\":[', ''),']}]', '') AS prediction,\n",
        "  FROM step1\n",
        "  JOIN UNNEST(step1.instance_list) AS instance\n",
        "  WITH OFFSET AS f1\n",
        "  JOIN UNNEST(step1.prediction_list) AS prediction\n",
        "  WITH OFFSET AS f2\n",
        "  ON f1=f2\n",
        "),\n",
        "\n",
        "step3 AS\n",
        "(\n",
        "  SELECT \n",
        "    model, \n",
        "    model_version, \n",
        "    time,\n",
        "    @json_features_extraction,\n",
        "    SPLIT(prediction, ',') AS class_probabilities, \n",
        "  FROM step2\n",
        "),\n",
        "\n",
        "step4\n",
        "AS\n",
        "(\n",
        "  SELECT * EXCEPT(class_probabilities)\n",
        "  FROM step3\n",
        "  JOIN UNNEST(step3.class_probabilities) AS class_probability\n",
        "  WITH OFFSET AS class_index\n",
        "),\n",
        "\n",
        "step5\n",
        "AS\n",
        "(\n",
        "  SELECT\n",
        "    model,\n",
        "    model_version,\n",
        "    time,\n",
        "    @input_features,\n",
        "    @class_probability_pivoting,\n",
        "    MAX(CAST(class_probability AS FLOAT64)) as prob_max\n",
        "  FROM step4\n",
        "  GROUP BY\n",
        "    model,\n",
        "    model_version,\n",
        "    time,\n",
        "    @input_features\n",
        ")\n",
        "\n",
        "SELECT\n",
        "  model,\n",
        "  model_version,\n",
        "  time,\n",
        "  @input_features,\n",
        "  @class_prob,\n",
        "  CASE\n",
        "  @case_conditions\n",
        "  END as predicted_class\n",
        "FROM\n",
        "  step5\n",
        "\n",
        "'''"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qWIlgsxRI4wV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sql_script = sql_script.replace(\"@project\", PROJECT_ID)\n",
        "sql_script = sql_script.replace(\"@dataset_name\", BQ_DATASET_NAME)\n",
        "sql_script = sql_script.replace(\"@table_name\", BQ_TABLE_NAME)\n",
        "sql_script = sql_script.replace(\"@view_name\", view_name)\n",
        "sql_script = sql_script.replace(\"@model_name\", MODEL_NAME)\n",
        "sql_script = sql_script.replace(\"@version\", VERSION_NAME)\n",
        "sql_script = sql_script.replace(\"@input_features\", input_features)\n",
        "sql_script = sql_script.replace(\"@json_features_extraction\", json_features_extraction)\n",
        "sql_script = sql_script.replace(\"@model_output_key\", MODEL_OUTPUT_KEY)\n",
        "sql_script = sql_script.replace(\"@class_probability_pivoting\", class_probability_pivoting)\n",
        "sql_script = sql_script.replace(\"@class_prob\", class_prob)\n",
        "sql_script = sql_script.replace(\"@case_conditions\", case_conditions)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sSZOugk-XmW0",
        "colab_type": "text"
      },
      "source": [
        "Print the generated script"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lm3YX_QrXmfI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(sql_script)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7_ArYsvKqn3j",
        "colab_type": "text"
      },
      "source": [
        "### 6.2. Create a view to parse the logs"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-ZwFA1OGUMPz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "client.query(query = sql_script)\n",
        "print(\"View was created or replaced.\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E6qXk_QjUMHq",
        "colab_type": "text"
      },
      "source": [
        "### 6.3. Query the view"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u9WDwhQbU0qX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "query = '''\n",
        "  SELECT * FROM \n",
        "  `{}.{}` \n",
        "  LIMIT {}\n",
        "'''.format(BQ_DATASET_NAME, view_name, 3)\n",
        "\n",
        "pd.io.gbq.read_gbq(\n",
        "    query, project_id=PROJECT_ID).T"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FknjMevMLyN3",
        "colab_type": "text"
      },
      "source": [
        "# Cleaning up\n",
        "\n",
        "To clean up all GCP resources used in this project, you can [delete the GCP\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial."
      ]
    }
  ]
}